Skip to content

fix(experiment): make run identity collision-safe - #20

Open
rschumann wants to merge 4 commits into
SantanderAI:mainfrom
rschumann:fix/collision-safe-run-identity
Open

fix(experiment): make run identity collision-safe#20
rschumann wants to merge 4 commits into
SantanderAI:mainfrom
rschumann:fix/collision-safe-run-identity

Conversation

@rschumann

Copy link
Copy Markdown

What this fixes

Closes #19.

ExperimentRunner.run_single() decided whether a run had already completed from
an identifier built only from model alias, regime, condition and seed. Changing
the case count — or the resolved model, the distributions configuration, or a
provider parameter — produced the same identifier, so skip_completed=True
could treat a materially different experiment as already done and return None
without executing it.

10 cases   -> mock-R2-S0-seed42
1000 cases -> mock-R2-S0-seed42   # silently skipped

Approach

The readable label and machine identity are now separate:

Field Role
run_label the previous readable string, unchanged
identity_sha256 SHA-256 over a versioned canonical payload — decides resume
identity the canonical payload, so the digest is reproducible
run_id the previous readable identifier, unchanged for compatibility

Identity covers the case count, regime, condition, seed, a digest of the
resolved model ID, a digest of the parsed distributions configuration,
declared adapter/deployment identity, constructor-configured provider generation
parameters (top_p for the current OpenAI-compatible and Bedrock adapters), and
the package version. Per-call temperature and max_tokens are selected by
governance-regime code and covered by that package version. The distributions
fingerprint is semantic, so comments and formatting do not invalidate earlier
runs while changing a parameter does.

New standard-library-only module src/mech_gov/experiment/identity.py owns
canonicalization, digests, stored-record validation and safe diffs.
It never reads the environment, touches the filesystem, invokes a model, or
accepts raw provider configuration.

Failing open

Runs that cannot prove equivalence always execute rather than risk a wrong skip:
no stable non-empty model_id, undeclared or invalid provider identity,
ambiguous credentialed endpoint, unavailable package version, or unreadable
completion state. Malformed JSONL records, including integers rejected by
Python's conversion-size limit, are ignored rather than aborting the run.
Provider identity hook failures are isolated from experiment execution. A
sentinel never makes two unknown configurations compare equal.
Recomputing a run costs time; wrongly skipping one silently corrupts a result
set.

Backward compatibility

  • Legacy records carrying only run_id are read, never rewritten or
    deleted
    , and never treated as equivalent to a versioned identity — a
    matching label logs MGI001 and the run executes.
  • Result model_id, the manifest, both JSONL output paths, and the
    on_run_complete callback are unchanged.
  • run_id and CLI display retain the legacy readable value. New machine
    identity is additive and never parsed from run_id.
  • LLMInterface.resume_identity_parameters() is a concrete method returning
    None, so existing custom providers keep working; they are simply treated as
    non-resumable.
  • Unknown schemas, malformed digests and payload mismatches warn and execute.

Warnings use stable codes MGI001MGI008 on the
mech_gov.experiment.identity logger, naming the fields that differ and using a
12-hex SHA-256 reference for correlation. They never print raw labels, paths, or
configuration values; MGI008 distinguishes unreadable completion state from
invalid stored identity.

OpenAI-compatible endpoint targets and Bedrock regions are represented only by
SHA-256 fingerprints. These are pseudonymous identifiers, not encryption.
SageMaker stays non-resumable because endpoint name plus region does not prove
which account-scoped deployment is selected. Existing raw model_id fields
remain unchanged for compatibility. The readable model alias also remains
public in run_id, run_label and identity; it must not contain credentials
or private identifiers.

Verification

  • 307 tests pass on Python 3.10, 3.11, and 3.12 with 96% total coverage and
    100% coverage on experiment/identity.py.
  • The pinned canonical byte/digest vector and ambient-decimal-context
    regression pass on all three supported interpreters.
  • Ruff, Black, Mypy, the offline quickstart, governance CLI and package build
    pass.
  • Dependency audit, dependency-license allowlist, SPDX headers, the upstream
    internal-pattern scan, changed-code Bandit comparison, tracked-tree secret
    scan and private-project/path scan pass.
  • The built wheel installs and imports successfully.
  • No dependency is added.

Windows was not executed; no platform-specific code is introduced.

Known provenance boundary: identity uses the package version. Source edits made
without a version change are outside the equivalence contract; editable-checkout
development should use skip_completed=False.

Questions for maintainers

  1. Code provenance. Identity uses the package version. Git commit stays
    traceability-only in the existing code_version field, because
    _get_code_version() runs git rev-parse in the caller's working directory
    and therefore reports whatever repository the user happens to run from. Would
    you prefer a stricter contract, accepting that runs from a source checkout
    become non-resumable?
  2. Provider parameters. Adding resume_identity_parameters() widens
    LLMInterface. If you would rather not, I can drop it and narrow the v1
    guarantee to RunConfig plus distributions and model identity, documenting
    the residual collision risk.
  3. SageMaker identity. It is non-resumable by default because the current
    configuration lacks an account-scoped immutable deployment identifier. Would
    you prefer an explicit safe deployment-ID field in a later change?
  4. Legacy compatibility mode. The selected safe default always reruns
    label-only legacy records because they cannot prove equivalence. Would you
    prefer a clearly named opt-in flag that restores label-based legacy resume
    while explicitly accepting collision risk?

ExperimentRunner decided whether a run had already completed using an
identifier built only from model alias, regime, condition and seed. Changing
the case count, the resolved model, the distributions configuration or a
provider parameter produced the same identifier, so skip_completed=True could
treat a materially different experiment as already done and return None
without executing it.

Separate the readable label from machine identity. run_label keeps the previous
readable string; identity_sha256 is a SHA-256 over a versioned, allowlisted
canonical payload and is the only thing that decides resume. run_id becomes
"<run_label>--<12 hex>" and stays an opaque string for existing consumers.

Identity covers the case count, regime, condition, seed, a digest of the
resolved model id, a digest of the parsed distributions configuration, provider
generation parameters and the package version. The distributions fingerprint is
semantic, so comment and formatting changes do not invalidate earlier runs
while a changed parameter does.

Runs that cannot prove equivalence -- no stable model id, undeclared provider
parameters, unavailable package version -- are non-resumable and always execute
with a warning. Failing open costs compute; failing closed silently corrupts a
result set.

Legacy records carrying only run_id are read but never rewritten and never
treated as equivalent to a versioned identity. Unknown schemas, malformed
digests and payload mismatches warn and execute. Warnings use stable codes
MGI001-MGI007 on the mech_gov.experiment.identity logger and name the fields
that differ without printing their values.

Add LLMInterface.resume_identity_parameters(), returning None by default so
existing custom providers keep working and are simply non-resumable.

95 new tests; no dependency added.
@rschumann
rschumann requested review from a team as code owners July 29, 2026 14:28
@github-actions

github-actions Bot commented Jul 29, 2026

Copy link
Copy Markdown

All contributors have signed the CLA ✍️ ✅
Posted by the CLA Assistant Lite bot.

@rschumann

Copy link
Copy Markdown
Author

I have read the CLA Document and I hereby sign the CLA

github-actions Bot added a commit that referenced this pull request Jul 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Experiment resume can skip changed configurations due to run_id collisions

1 participant